Response to office action of 6/24/10 for appl. 10/780,500 

Remarks 

Claims 1-3, 5-11, 13-28 and 32-41 are pending in this application. 

Claims 22-28, 32-35, 38, and 40-41 stand rejected under 35 U.S.C. 
103(a) as being unpatentable over Szeliski et al. (US Patent Number 6,157,747, 
herein referred to as Szeliski) in view of Seago (US Patent Number 5,990,900). 
Claims 11, 13-21, 23, and 36-37 stand rejected under 35 U.S.C. 103(a) as 
being unpatentable over the combination of Szeliski and Seago, and in further 
view of Blank (US Patent Number 5,469,536). Claims 1-3, 5-6, 8-10 and 39 
stand rejected under 35 U.S.C. 103(a) as being unpatentable over Szeliski in 
view of Luken (US Patent Number 5,923,334), and in further view of Seago. 
Claim 7 stands rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Szeliski, Luken, and Seago, and in further view of Blank. 

Applicant wishes to thank Examiner Prendergast for the telephonic 
interview between the Examiner and Applicant's attorney, John Conway, on 
December 7, 2010. During the interview, the method taught by Seago '900 for 
modeling objects in 2D images was discussed. In particular, the applicability 
of Seago '900's teachings for preparing a 3D model of an object when no 2D 
image in a set of 2D perspective images contains the complete object was 
discussed. Attorney Conway maintained that Seago 900's disclosure does not 
teach steps for creating accurate 3D models from each 2D perspective image in 
a set of images when each 2D image leaves out a portion of the object. 
Examiner Prendergast pointed to passages from Seago '900 that she 
maintained indicated the applicability of Seago 's method to this situation. 
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Attorney Conway stated the response would address this situation in more 
detail. This situation is addressed at page 26 and in appendix 1, below. 
Agreement was not reached during the interview. 

Claim Rejections - 35 U.S. C g 103(a) 
Claims 22-28, 32-35, 38, and 40-41 stand rejected under 35 U.S.C. 
103(a) as being unpatentable over Szeliski et al. (US Patent Number 6,157,747, 
herein referred to as "Szeliski") in view of Seago (US Patent Number 5,990,900, 
herein referred to as "Seago"). 

The rejection of Claims 22-28, 32-35, 38, and 40-41 for obviousness over 
Szeliski in view of Seago must fail because neither Szeliski nor Seago teaches 
creating a geometric model of an object within an image panorama including 
identifying at least one boundary of the object where the object occupies a field 
of view in the panorama of greater than 180 degrees, as required by each of 
Claims 22-28, 32-35, 38, and 40-41. Details follow. 
Claim 22 requires in pertinent part: 

"(a)... receiving instructions from a user identifying a three-dimensional 
geometric surface within an image panorama, the image panorama containing 
an object having one or more textures, the object occupying a field of view of 
more than 180 degrees in the panorama; 

(b) using the computer creating a geometric model of the image 
panorama based at least in part on the three-dimensional geometric surface 
and the directional vector, wherein creating a geometric model includes 
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identifying at least one boundary of the object and using the identified 
boundary to associate geometry information with the object, the geometry 
information comprising 3-D coordinates describing the position and orientation 
of the object boundary in a reference coordinate system;... (annotations added). 

Likewise, independent Claims 32 and 38 include similar limitations to 
the cited limitations of Claim 22. 

The subject office action admits that: 

"Szeliski et al. does not specifically teach wherein creating a geometric model 
includes identifying at least one boundary of the object and using the identified 
boundary to associate geometry information with the object, the geometry 
information comprising 3-D coordinates describing the position and orientation 
of the object boundary in a reference coordinate system." (See, Office Action, 
page 21.) 

Thus, if the combination of Szeliski and Seago teaches limitation (b) of 
Claim 22, Seago '900 must supply the teaching of modeling the object using an 
identified boundary of the object where the object occupies a field of view of 
more than 1 80 degrees in the panorama as required by limitation (a) of Claim 
22. However, Seago's method fails when the object to be modeled occupies 
a field of view greater than 180 degrees in the input image(s). Details 
follow: 

The input to Seago's method is a 2-D perspective image or a group of 2-D 
images displaying the object to be modeled. (See, e.g., Seago, col. 11, lines 45- 
47). However: 
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(1) When the object to be modeled requires a field of view of more 
than 180 degrees, the entire object will not fit onto a single 2D 
perspective image. Each 2D perspective image in the set of 
images can show only a portion of the object; and 

(2) Seago's method for modeling an object from multiple 2D 
perspective images requires that a separate 3D model be made of 
the object from each of the 2D perspective images. When each 
2D image does not contain the entire object, Seago's method fails 
because the required 3D models of the object cannot be created. 
(See, Seago '900 col. 7, lines 29-33; fig. 6A, step 124; and col. 7 
lines 50-53.) 

Thus, Seago '900's method of 3D modeling will fail for any object that covers a 
field of view greater than 1 80 degrees in the input panorama and the 
combination of Seago '900 with Szeliski '747 will not satisfy the limitations of 
Claim 22 of the subject application. 

Statements 1 and 2 concerning Seago's method are illustrated in figs. 1- 
12, which are shown below. 
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1. Because the object requires a field of view of more than 180 degrees, 
the object will not fit on a single 2D perspective image. Each 2D image in 
the set of images will show only a portion of the object. 




Fig. 1 

The curved wall is an object that extends beyond 1 80 degrees in the input 
panoramic image. The lollipop represents the camera position; the wall object 
surrounds the camera position. 
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Fig. 2. 

The camera acquires 2D perspective images. To create a mosaic similar 
to Szeliski, the user must take multiple overlapping photos to span the wall's 
extent, which is > 180 degrees. 
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Fig. 4. 
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Fig. 5. 

Without loss of generality, the mosaiced image to be used comprises three 2D 
images that have sufficient overlap to register the images. 
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Fig. 6. 

But the final mosaic created with the three initial images cannot be 
displayed as a single 2D perspective image that preserves the features that are 
used to register or model the scene and objects within the scene. As shown 
above, using Szeliski's method, the image pose cannot change without 
distorting the original images or cropping features not visible within the 1 80 
degree limit for the 2D perspective image required as input for Seago's method. 
The white portions of the images of the wall object illustrate that at least a 
portion of the wall object will be left out of any single 2D perspective image. 
After all, the wall object spans a field of view with more than 180 degrees. 
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Warping and remapping a wall object that spans a field of view with more than 
1 80 degrees so that it "fits" on a single 2D image would yield an image similar 
to fig. 7: 




Fig. 7. 

The warping needed to force objects with extents > 180 degrees onto a single 
2D image destroys the perspective view of these objects. Such warping renders 
the single image useless for Seago's method, which demands perspective 
images of objects as inputs. (See, e.g., Seago fig. 2. In particular, steps 42, 44, 
46, and 48 cannot be performed with a warped image.) To maintain 
perspective, such an object must be spread across multiple perspective images. 



2. Seago's method for modeling an object from multiple 2D perspective 



images requires that a 3D model be made of the object from each of 
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the multiple 2D perspective images. If each 2D image does not contain 
the entire object, Seago's method fails because the required 3D models 
of the object from each image cannot be created. (See, Seago '900 col. 
7, lines 29-33; fig. 6A, step 124; and col. 7 lines 50-53.) 



Seago 900's method of modeling a 3D object from a 2D image is described at 
col. 4, line 34 through col. 6, line 35. Fig. 2 of Seago '900 is a flow chart that 
shows Seago's steps for creating the 3D model. Steps 42 and 44 in fig. 2 
comprise: 



"The approximate orientation determination involves visualizing 
the selected object and determining the location of the selected 
object's three orthogonal sides or in other words the object's 
natural coordinate system. Using the perspective image, the user 
approximates the location of the vanishing points of the selected 
object within the image plane. The first step in the determination 
of vanishing point locations is at block 44, where lines are created 
that overlap parallel lines on the selected object or other objects 
within the digital image that appear to have the same orientation 
(i.e., have the same vanishing points and natural coordinate 
system) as that of the selected object based on the approximate 
orientation determination. Line creation can be performed 
automatically by image analysis software, which determines edges 
of objects and creates lines overlapping the determined edges. 
More preferably, the user uses line generating software to 
designate these lines. The lines overlapped on the selected object 
are lines that lie parallel to one another on the actual object 
represented by the image of the selected object. On the displayed 
two-dimensional perspective image, the overlapped lines do not 
appear to be parallel, but appear to converge at varying angles to 
the vanishing points. The minimum number of lines required 
for each of the three vanishing points for this designation step 
is two. Line designation is illustrated in FIG. 7 and described in 
more detail below. Following line designation, the system 
calculates vanishing point..." (Seago '900, col. 4, line 53 to col. 5, 
line 12.) 
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Clearly, if any of the object's three orthogonal sides cannot be located in 
each image, steps 42 and 44 cannot be performed. Using the simplest example 
of a wall that does not fit on a single 2D image, consider the process of 
geometrically modeling the wall, which is taken as two overlapping images, A 
and B. (It is easy to construct a more extreme example of how Seago's method 
must fail when the object will not fit entirely on a single 2D image — such as a 
wall object that spans more than 180 degrees.) 



To model a plane, four points or edges need to be corresponded between 
images A and B. However, because image A contains only a portion of the wall, 
points 3 and 4 that describe the wall's rightward extent must be selected 
arbitrarily, as shown in figs. 9. and 10: 




A 



B 



Fig. 8. 
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? 



Fig. 10 

Because no features are visible for vertices 3 and 4 for the wall object in image 
A, the slope and position of line 3-4 cannot be determined unambiguously. 
(The questions marks in fig. 10 denote this uncertainty.) 
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Thus, steps 42 and 44 in fig. 2 of Seago '900 cannot be performed accurately 
for image A for the rightmost orthogonal side of the wall object. Thus, Seago's 
method will fail to produce a 3D model of the wall from image A. 

Likewise, in image B, vertices 3 and 4 are visible in the image and can be 
determined with certainty, but vertices 1 and 2 are arbitrary as shown in figs. 
11 and 12: 




Fig. 11. 
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Fig. 12. 

Because no features are visible for vertices 1 and 2 for the wall object in image 
B, the slope and position of line 1-2 cannot be determined unambiguously. 
(The questions marks in fig. 12 denote this uncertainty.) Steps 42 and 44 in 
fig. 2 of Seago '900 cannot be performed accurately for image B for the 
rightmost orthogonal side of the wall object. Thus, Seago's method will fail to 
produce a 3D model of the wall from image B. 

Thus, Seago's method will fail because separate 3D models cannot be 
created from each of the multiple 2D perspective images needed to show an 
object with extent > 180 degrees. 

Seago '900 only addresses generating a single three-dimensional object 
from two or more three-dimensional objects created from two-dimensional 
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images of a single object at col. 7, line 29 through col. 8, line 42; fig. 6A and 
6B; and col. 11, lines 45-47. Please refer to appendix 1, attached below, for 
a discussion showing that these portions of Seago '900 do not remedy the 
deficiency in creating an accurate 3D model from multiple incomplete 2D 
perspective image that was described above. 

To be sure, Szeliski '774's method can be used to produce a single 2D 
image that has four vertices visible to model in the case of the wall shown in 
figs. 8-12, since the wall's extent is not greater than 180 degrees. But for 
objects that require more than a 180 degree field of view, the necessary vertices 
(and edges/planes) cannot be projected onto a single 2D perspective image to 
satisfy Seago '900's method. (See figs. 1-7 and accompanying discussion 
above.) If a mosaic/ panorama with an object extends beyond 180 degrees, the 
image needs to be warped/ remapped to fit into a single 2D image. Warping or 
remapping features does not work for Seago 's method because Seago 's method 
relies on perspective, e.g., straight lines on objects are no longer straight, etc. 

Because step "b" of Claims 22, 32 and 38, for objects that occupy a field 
of view greater than 180 degrees as required by step "a," is neither taught nor 
suggested by any combination of the teachings of Szeliski '774 and Seago '900, 
a prima facie case of obviousness is lacking. Claims 22, 32 and 38 are 
therefore deemed patentable. Claims 23-28, 33-35 and 40-41 depend from 
Claims 22, 32, and 38, respectively and add further limitations. Each of these 
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claims is deemed nonobvious over Szeliski in view of Seago for at least the 
same reasons as for Claims 22, 32 and 38. 

Claims 11, 13-21, 23, and 36-37 stand rejected under 35 U.S.C. 103(a) 
as being unpatentable over the combination of Szeliski and Seago, and in 
further view of Blank (US Patent Number 5,469,536). 

The rejections of Claims 11, 13-21, 23, and 36-37 for obviousness rely on 
Szeliski and Seago for teaching the limitations cited above for Claim 22 and 32, 
namely: limitation (b) of Claim 22, where the object occupies a field of view of 
more than 1 80 degrees in the panorama as required by limitation (a) of Claim 
22. Therefore, a prima facie case of obviousness is lacking for each of these 
claims because, as shown above, Szeliski and Seago do not teach, disclose or 
suggest these limitations of the claims. Claims 11, 13-21, 23, and 36-37 are 
therefore deemed patentable over the cited references. 

Claims 1-3, 5-6, 8-10 and 39 stand rejected under 35 U.S.C. 103(a) as 
being unpatentable over Szeliski in view of Luken (US Patent Number 
5,923,334), and in further view of Seago. 

The rejections of Claims 1-3, 5-6, 8-10 and 39 for obviousness rely on 
Szeliski and Seago for teaching the limitations cited above in rejecting Claim 22 
and 32, namely: limitation (b) of Claim 22, where the object occupies a field of 
view of more than 1 80 degrees in the panorama as required by limitation (a) of 
Claim 22. Therefore, a prima facie case of obviousness is lacking for each of 
these claims because, as shown above, Szeliski and Seago do not teach, 
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disclose or suggest these limitations of the claims. Claims 1-3, 5-6, 8-10 and 
39 are therefore deemed patentable over the cited references. 

Claim 7 stands rejected under 35 U.S.C. 103(a) as being unpatentable 
over the combination of Szeliski, Luken, and Seago, and in further view of 
Blank. 

The rejection of Claim 7 for obviousness relies on Szeliski and Seago for 
teaching the limitations cited above in rejecting Claim 22 and 32, namely: 
limitation (b) of Claim 22, where the object occupies a field of view of more than 
180 degrees in the panorama as required by limitation (a) of Claim 22. 
Therefore, a prima facie case of obviousness is lacking for Claim 7 because, as 
shown above, Szeliski and Seago do not teach, disclose or suggest these 
limitations of the claims. Claim 7 is therefore deemed patentable over the cited 
references. 
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Applicant requests reconsideration of all pending claims and a notice of 
allowance. The Examiner is requested to telephone the undersigned if any 
matters remain outstanding so that they may be resolved expeditiously. The 
Commissioner is hereby authorized to charge any deficiency in the fees filed, 
asserted to be filed or which should have been filed herewith to our Deposit 
Account No. 19-4972. 

Respectfully submitted, 
/John L. Conway, #48,241/ 
John L. Conway 

Sunstein, Kann, Murphy & Timbers LLP Registration No. 48,241 
125 Summer Street Attorney for Applicant 

Boston, MA 02110-1618 
(617) 443-9292 

02894/00108 1366345.1 



Page 29 of 33 



Response to office action of 6/24/10 for appl. 10/780,500 



Appendix 1 

Seago '900's Method's Deficiency for Modeling an Object from Multiple 
Incomplete 2D Perspective Images. 

Seago '900 does not teach the steps for creating an accurate 3D object 
model from multiple 2D perspective images, unless each 2D image contains the 
complete object. 

Seago '900 addresses generating a single, 3D object from multiple two- 
dimensional images of a single object only at the following passages: col. 7, line 
29 through col. 8, line 42; fig. 6A and 6B; and col. 11, lines 45-47. Thus, if 
Seago' 900 teaches creating a 3D model from multiple incomplete 2D images, 
the teaching must be in those passages. Analysis of these passages shows that 
these passages do not provide such a teaching. 

Seago 's explanation begins by stating: 

"FIG. 6A illustrates the process by which the image converting system 
20 shown in FIG. 1 generates a single three-dimensional object from two 
or more three-dimensional objects created from two-dimensional images 
of a single object, as shown in FIGS. 2-4 (sic)." (Col. 7, lines 29-33). 

Seago then refers to blocks 120 and 122 of fig 6A to require that relative 

orientation information between each 2D perspective view be determined: 

"At block 120, a sufficient set of features or significant features are 
identified on each two-dimensional object in the multiple views. A 
sufficient set of features may include any set of matching or conjugate 
vertices, vanishing lines or planes between the multiple images. The set 
is sufficient when the conjugate matches between the images provides 
enough information for determining relative orientation information 
between each view. 

Relative orientation for each view is determined at block 122. The 
relative orientation of a view is how the vertices, vanishing lines or 
planes it sees correspond to those of the other views. For example, if 
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two views of the same object differ by 180. degree., the left vanishing 
lines and planes in one view are the right vanishing lines and planes in 
other view." (Col. 7, lines 33-47). 

When each 2D image does not contain a complete object - as illustrated in figs. 

8-12 above - a complete set of matching points do not exist because of the 

ambiguities introduced by the incomplete image. Seago does not address a 

remedy for this situation. The example of views differing by 180 degrees and 

that "left vanishing lines and planes in one view are right vanishing lines and 

planes in the other" merely describes one example of disparate views of the 

same object, but do not address the fundamental problem of insufficient 

conjugate points between incomplete objects in the views. 



Seago then goes on to describe creating a single 3D object model from 3D 
models created from each of the multiple 2D images at block 124 of fig. 6A: 



"At block 124, the single three-dimensional object is created in world 
coordinate space based on the three-dimensional coordinate spaces 
(SWCS) for each image, and the objects in each two-dimensional image 
and the determined relative orientations. The origin of the world 
coordinate space is set at a default position and the orientation is set to 
a default orientation. If components of the created three-dimensional 
object are inverted or out of place because of ambiguous or improper 
relative orientation determination, the user can manually correct the 
created object by using the user interface. 

The three-dimensional object generation is performed according to the 
process illustrated in FIG. 6B and described in more detail below..." 
(Col. 7, lines 50-62). 

Seago then describes the process of generating a single 3D object from 
multiple 3D objects. The basic process of modeling each object is as shown in 
figs 2-4 and applied as follows: 
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"FIG. 6B illustrates the three-dimensional object generation process 
from block 124 of FIG. 6A. 

First, at block 128, planes referenced to the world coordinate space are 
set at a default position, each image's viewpoint is set at a default 
position in world coordinate space and each image's viewpoint is set to a 
default focal length and orientation. 

Next, at block 130, the intersection points of the set planes which 
correspond to vertices in the created three-dimensional object are 
projected onto each image's image plane using each image's set 
viewpoint focal length. 

At block 132, the projected vertices' two-dimensional values on the 
image planes are determined. 

Then, at block 134, the image plane location of each designated vertex of 
the selected object is compared to its corresponding determined 
projected vertex. An error function is determined based on the 
comparisons. See block 136 

Next, at block 138, an iterative non-linear numerical optimization 
algorithm is executed according to the determined error function. More 
specifically, at decision block 140, the non-linear optimization algorithm 
determines if the determined error function is at an acceptably low level. 
An acceptable error function level exists when the difference between the 
projected vertices and the selected object's vertices is below a minimal 
value. 

If the determined error function is at an acceptably low level, the process 
continues as shown in FIG. 6A. However, if the non-linear optimization 
algorithm determines that the determined error function is not at an 
acceptably low level, the planes' positions and each image's position in 
world coordinate space, and each image's viewpoint focal length and 
orientation are adjusted based on the determined error function. See 
block 142. The planes are adjusted by moving each plane along the 
world coordinate space axis that is parallel to the plane's normal vector. 
The viewpoints' positions are adjusted by moving each viewpoint within 
the world coordinate system. The viewpoints' orientations are adjusted 
by orienting each viewpoint with respect to the world coordinate system. 
Focal length of each images' viewpoint is adjusted along the normal from 
the center of its respective image. These adjustments change the 
position of all projected vertices, thereby changing the ensuing 
comparison. After adjustment has taken place, the steps performed in 
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blocks 130-136 are repeated until an acceptable three-dimensional 
model is found. Three-dimensional object generation is illustrated by an 
example shown in FIG. 12 and described below." (Col. 7, lines 66 to col. 
8, line 42). 

Steps 128 - 140 of fig. 6B address adjusting positions of planes and 
viewpoints in the world coordinate system and the viewpoints parameters so 
that the projected vertices on the 2D images and the object vertices are 
acceptably close to each other. These steps, however, say nothing about 
creating a single complete 3D object model from two or more incomplete 3D 
models. For example, no mention is made of filling the gaps in any single 3D 
model that arise from the incomplete 2D images. 



Seago finishes the discussion of creating a single 3D object model as follows: 

As will be readily appreciated by those of ordinary skill in the art of 
graphics processing and manipulation, various graphical tools and 
computer system components may be used to perform many of the user 
interactions described above. While FIGS. 7-13 show an implementation 
of the invention using only a single input image, the system can also use 
two or more input images. In this case the steps illustrated in FIGS. 6A 
and 6B and described above are applied to all images . (Col. 7, lines 29- 
33). 

Clearly, Seago '900 provides no teaching that would enable one of 
ordinary skill in the art to create a single 3D model from two or more 
incomplete 3D models created from two or more input 2D perspective images 
that do not contain the complete object. If the Examiner disagrees, Applicant 
respectfully requests that the Examiner state for the record specifically where 
such teachings are found in Seago '900. 
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